process_market_breadth.py

Overview

process_market_breadth.py is a Phase 4 analytics script that calculates Relative Strength (RS) scores for all stocks and generates comprehensive sector and industry-level breadth metrics for the Market Breadth Dashboard.

Pipeline Position: Phase 4 - Analytics generationCritical Function: Powers the multi-tab Market Breadth Dashboard with Moving Average, RS Rating, and Near-High analytics

Purpose

This script performs two major functions:

RS Score Calculation: Computes weighted Relative Strength ratings (1-99) for all stocks
Breadth Analytics: Generates sector and industry-level breadth metrics across three dimensions:
- Moving Average breadth (SMA 200/50/20)
- RS Rating distribution (RS 70/80/90)
- Near 52-Week High distribution (1%/2%/5%)

Input Files

all_stocks_fundamental_analysis.json

JSON

required

Master stock database with price returns and technical indicators including:

1 Year Returns(%), 6 Month Returns(%), 3 Month Returns(%), 1 Month Returns(%)
SMA Status (e.g., “Above SMA 200, Above SMA 50”)
% from 52W High
Sector and Basic Industry classifications

Output Files

all_stocks_fundamental_analysis.json

JSON

Updated master file with new RS Rating field added to each stock (1-99 scale)

sector_analytics.json

JSON

Comprehensive sector and industry breadth metrics for dashboard renderingStructure:

{
  "Sectors": [
    {
      "Type": "Sector",
      "Name": "Technology",
      "StockCount": 89,
      "Breadth_SMA200": 72.1,
      "Breadth_SMA50": 65.3,
      "Breadth_SMA20": 58.4,
      "Breadth_RS70": 45.2,
      "Breadth_RS80": 28.1,
      "Breadth_RS90": 12.4,
      "NearHigh_1pc": 8.9,
      "NearHigh_2pc": 15.7,
      "NearHigh_5pc": 31.5
    }
  ],
  "Industries": [
    {
      "Type": "Industry",
      "Name": "Software Development",
      "ParentSector": "Technology",
      "StockCount": 34,
      "Breadth_SMA50": 70.6,
      "Breadth_RS80": 32.4,
      "Breadth_NH5": 38.2,
      "Contribution_SMA50_%": 26.9,
      "Contribution_RS80_%": 12.4,
      "Contribution_NH5_%": 14.6
    }
  ]
}

Processing Logic

1. RS Score Calculation

Calculates a weighted Relative Strength score using multiple timeframes:

def calculate_rs_score(df):
    """Calculates Relative Strength Score (1-99) for each stock."""
    w_1y, w_6m, w_3m, w_1m = 0.4, 0.2, 0.2, 0.2
    
    for col in ['1 Year Returns(%)', '6 Month Returns(%)', '3 Month Returns(%)', '1 Month Returns(%)']:
        df[col] = pd.to_numeric(df[col], errors='coerce').fillna(0)
    
    df['RS_Raw'] = (df['1 Year Returns(%)'] * w_1y) + \
                   (df['6 Month Returns(%)'] * w_6m) + \
                   (df['3 Month Returns(%)'] * w_3m) + \
                   (df['1 Month Returns(%)'] * w_1m)
    
    if not df.empty:
        # Professional RS Percentile: 1-99
        df['RS Rating'] = (df['RS_Raw'].rank(pct=True, method='min') * 99).apply(np.ceil).astype(int)
        df['RS Rating'] = df['RS Rating'].clip(1, 99)
    else:
        df['RS Rating'] = 0
    return df

Weighting:

1 Year Returns: 40%
6 Month Returns: 20%
3 Month Returns: 20%
1 Month Returns: 20%

Output Range: 1-99 (percentile rank)

2. Breadth Flags Calculation

Pre-calculates boolean flags for efficient aggregation:

def is_above(status_str, sma_key):
    if not isinstance(status_str, str): return False
    return sma_key in status_str and "Above" in status_str

# PRE-CALCULATE FLAGS
df['Above_SMA_200'] = df['SMA Status'].apply(lambda x: is_above(x, "SMA 200"))
df['Above_SMA_50'] = df['SMA Status'].apply(lambda x: is_above(x, "SMA 50"))
df['Above_SMA_20'] = df['SMA Status'].apply(lambda x: is_above(x, "SMA 20"))

# Near High Buckets
df['Dist_High'] = df['% from 52W High'].apply(lambda x: abs(x) if x is not None else 999)

# RS Buckets
df['Above_RS_70'] = df['RS Rating'] >= 70
df['Above_RS_80'] = df['RS Rating'] >= 80
df['Above_RS_90'] = df['RS Rating'] >= 90

3. Sector-Level Analytics

Generates breadth percentages for each sector:

sector_list = []
sector_groups = df.groupby('Sector')
for sector_name, group in sector_groups:
    if not sector_name or sector_name == "N/A": continue
    
    sector_list.append({
        "Type": "Sector",
        "Name": sector_name,
        "StockCount": len(group),
        "Breadth_SMA200": round(group['Above_SMA_200'].mean() * 100, 1),
        "Breadth_SMA50": round(group['Above_SMA_50'].mean() * 100, 1),
        "Breadth_SMA20": round(group['Above_SMA_20'].mean() * 100, 1),
        "Breadth_RS70": round(group['Above_RS_70'].mean() * 100, 1),
        "Breadth_RS80": round(group['Above_RS_80'].mean() * 100, 1),
        "Breadth_RS90": round(group['Above_RS_90'].mean() * 100, 1),
        "NearHigh_1pc": round((group['Dist_High'] <= 1.0).mean() * 100, 1),
        "NearHigh_2pc": round((group['Dist_High'] <= 2.0).mean() * 100, 1),
        "NearHigh_5pc": round((group['Dist_High'] <= 5.0).mean() * 100, 1)
    })

4. Industry-Level Analytics

Generates industry metrics with sector contribution calculations:

industry_list = []
industry_groups = df.groupby('Basic Industry')
for ind_name, group in industry_groups:
    if not ind_name or ind_name == "N/A": continue
    parent_sector = group['Sector'].iloc[0]
    sector_total = sector_counts.get(parent_sector, 1)
    
    # Absolute Breadth Contribution (Industry Winners / Sector Total Stocks)
    win_50 = group['Above_SMA_50'].sum()
    win_rs80 = group['Above_RS_80'].sum()
    win_nh5 = (group['Dist_High'] <= 5.0).sum()
    
    industry_list.append({
        "Type": "Industry",
        "Name": ind_name,
        "ParentSector": parent_sector,
        "StockCount": len(group),
        "Breadth_SMA50": round(group['Above_SMA_50'].mean() * 100, 1),
        "Breadth_RS80": round(group['Above_RS_80'].mean() * 100, 1),
        "Breadth_NH5": round((group['Dist_High'] <= 5.0).mean() * 100, 1),
        # Dashboard Style Contribution
        "Contribution_SMA50_%": round((win_50 / sector_total) * 100, 1),
        "Contribution_RS80_%": round((win_rs80 / sector_total) * 100, 1),
        "Contribution_NH5_%": round((win_nh5 / sector_total) * 100, 1)
    })

Fields Added to Master Stock Database

RS Rating

integer

Relative Strength rating on a 1-99 scale

Higher values indicate stronger relative performance
Calculated using weighted multi-timeframe returns
Percentile-based ranking across all stocks

Sector Analytics Fields

Breadth_SMA200

float

Percentage of stocks in sector trading above their 200-day SMA

Breadth_SMA50

float

Percentage of stocks in sector trading above their 50-day SMA

Breadth_SMA20

float

Percentage of stocks in sector trading above their 20-day SMA

Breadth_RS70

float

Percentage of stocks in sector with RS Rating >= 70

Breadth_RS80

float

Percentage of stocks in sector with RS Rating >= 80

Breadth_RS90

float

Percentage of stocks in sector with RS Rating >= 90

NearHigh_1pc

float

Percentage of stocks within 1% of 52-week high

NearHigh_2pc

float

Percentage of stocks within 2% of 52-week high

NearHigh_5pc

float

Percentage of stocks within 5% of 52-week high

Industry Contribution Metrics

Contribution_SMA50_%

float

(Industry stocks above SMA 50) / (Total sector stocks) * 100Measures the industry’s absolute contribution to sector strength

Contribution_RS80_%

float

(Industry stocks with RS >= 80) / (Total sector stocks) * 100

Contribution_NH5_%

float

(Industry stocks within 5% of high) / (Total sector stocks) * 100

Usage Example

python process_market_breadth.py

Expected Output:

Loading data for Multi-Tab Analysis (MA, RS, NH)...
Generating comprehensive analytics for all three dashboard tabs...
Success! Dashboard Engine complete for MA, RS, and Near-High views.

Dashboard Integration

The sector_analytics.json output powers three dashboard tabs:

Moving Average Tab: Uses Breadth_SMA200/50/20 metrics
RS Rating Tab: Uses Breadth_RS70/80/90 metrics
Near High Tab: Uses NearHigh_1pc/2pc/5pc metrics

Performance Notes

Market cap filter (300 Cr minimum) is currently disabled per user request
Temporary calculation columns are dropped before saving to reduce file size
All breadth percentages are rounded to 1 decimal place

bulk_market_analyzer.py - Provides input data with returns and technical indicators
process_historical_market_breadth.py - Generates time-series breadth data

Data Fetching Scripts

Processing Scripts

Standalone Scripts

Utilities

Overview

Purpose

Input Files

Output Files

Processing Logic

1. RS Score Calculation

2. Breadth Flags Calculation

3. Sector-Level Analytics

4. Industry-Level Analytics

Fields Added to Master Stock Database

Sector Analytics Fields

Industry Contribution Metrics

Usage Example

Dashboard Integration

Performance Notes

Data Fetching Scripts

Processing Scripts

Standalone Scripts

Utilities

​Overview

​Purpose

​Input Files

​Output Files

​Processing Logic

​1. RS Score Calculation

​2. Breadth Flags Calculation

​3. Sector-Level Analytics

​4. Industry-Level Analytics

​Fields Added to Master Stock Database

​Sector Analytics Fields

​Industry Contribution Metrics

​Usage Example

​Dashboard Integration

​Performance Notes

​Related Scripts

Overview

Purpose

Input Files

Output Files

Processing Logic

1. RS Score Calculation

2. Breadth Flags Calculation

3. Sector-Level Analytics

4. Industry-Level Analytics

Fields Added to Master Stock Database

Sector Analytics Fields

Industry Contribution Metrics

Usage Example

Dashboard Integration

Performance Notes

Related Scripts